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ABSTRACT 


Army  trainers  are  increasingly  turning  to  alternative  training  methods  to  exploit  low-cost,  technology-based 
solutions  to  fill  critical  training  gaps  and  increase  training  effectiveness.  One  technological  approach  that  has 
received  increased  interest  over  the  last  decade  is  game-based  training.  Army  agencies  and  organizations  have 
recognized  that  games  have  the  potential  to  augment  military  training  for  both  individuals  and  collectives.  However, 
compared  to  more  traditional  training  effectiveness  and  design  studies,  empirical  investigation  of  game-based 
training  is  in  its  infancy.  The  existing  body  of  research  is  only  starting  to  provide  insight  as  to  how  well  game-based 
training  works  relative  to  other  methods,  not  why  or  when  to  use  it.  Therefore,  the  purpose  of  this  research  was  to 
investigate  the  general  effectiveness  of  game-based  training  using  principles  generated  through  decades  of  research 
on  training  effectiveness.  Also,  the  research  literature  indicates  that  training  effectiveness  is  determined  by  the 
training  program  (e.g.,  the  game),  the  trainee  (e.g,,  personal  characteristics),  and  the  situational  context  of  the 
training  (Campbell  &  Kuncel,  2001).  Therefore,  this  study  also  focused  on  tire  conditions  under  which  game-based 
training  is  effective.  This  paper  describes  the  methods,  measures,  and  results  of  an  evaluation  with  165  Soldiers 
participating  in  game-based  training  as  units.  Pre-  and  post-training  measures  were  administered,  Results  indicate 
that  game-based  training  influenced  training  effectiveness.  Contextual  variables  such  as  level  of  preparation  prior  to 
the  training  also  influenced  the  effectiveness.  The  results  of  this  evaluation  provide  important  points  for  all  training 
designers  to  consider  when  using  game-based  training. 
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INTRODUCTION 

There  has  been  an  increased  interest  in  the  use  of 
serious  games,  or  the  use  of  games  for  non¬ 
entertainment  purposes  (e.g.,  training),  over  the  last 
decade,  Training  challenges  driven  by  an  increased 
diversity  of  operational  environments,  the  complexity 
of  possible  domestic  and  global  missions,  and 
competing  requirements  for  training  resources  have 
resulted  in  military  branches,  including  the  Army, 
turning  to  games  for  training.  Games  are  seen  as  an 
innovative,  low-cost  technology-based  solution  that 
can  rapidly  fill  critical  training  gaps  and  increase  the 
impact  and  effectiveness  of  training  for  our  Soldiers  in 
a  time  of  war. 

Many  Army  agencies  and  organizations,  including  the 
U.S.  Army  Training  and  Doctrine  Command 
(TRADOC),  have  recognized  that  games  have  the 
potential  to  augment  and  improve  military  training  for 
both  individuals  and  collectives.  Examples  of  active 
game-based  training  applications  include  Army 
DARWARS  Ambush  and  Tactical  Iraqi.  These  games 
are  being  used  to  train  Soldiers  on  a  variety  of  skills 
including  convoy  operations,  language,  cultural 
awareness,  and  other  tactical  drills  and  tasks. 

One  of  the  most  recent  game-based  training  efforts  is 
the  acquisition  and  fielding  of  Virtual  BattleSpaee2 
(VBS2):  U.S.  Army.  VBS2:  U.S.  Army  places  military 
trainees  in  realistic  urban  or  rural  settings  and  provides 
them  with  the  opportunity  to  practice  protocols  relating 
to  convoy  and  ambush  operations.  Trainees  can  drive 
Army  vehicles,  pilot  aircraft,  and  fire  weapons  across  a 
number  of  game  scenarios.  The  use  of  VBS2;  U.S. 
Army  is  designed  to  be  a  stepping  stone  in  Soldier 
training.  Ideally,  units  will  use  the  game  to  practice 
tactics,  techniques  and  procedures  prior  to  going  into 
the  field.  By  engaging  in  game-based  training,  Soldiers 
can  improve  their  skills  before  participating  in  a  live 
training  exercise. 

While  the  Army  has  established  a  game  technology 
program  of  record,  there  are  still  many  unanswered 
questions  regarding  game  technology  training 
effectiveness.  In  general,  game-based  training  is  only 
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beginning  to  be  held  to  the  rigorous  standards  that 
surround  more  conventional  training  programs.  During 
the  last  40  years,  there  have  been  a  total  of  seven 
articles  published  in  the  Annual  Review  of  Psychology 
on  training;  each  of  these  articles  serves  as  a  reminder 
of  the  progress  that  has  been  made  in  the  training  realm 
by  focusing  on  the  sound  science  behind  designing  and 
evaluating  effective  training  programs.  In  comparison, 
design  and  evaluation  research  focusing  on  game-based 
training  is  in  its  infancy.  With  the  increased  interest  in 
using  game-based  training,  it  would  bode  well  to  apply 
the  same  scientific  rigor  to  this  specific  training 
approach;  in  addition,  lessons  learned  from  the 
decades  of  research  on  conventional  training  programs 
can  be  examined  and  integrated  into  this  specific 
training  approach,  The  science  of  designing  and 
evaluating  training  programs  must  be  applied  to  game- 
based  training  if  game-based  training  is  going  to 
emerge  as  a  credible  and  effective  training  method. 

Initial  research  and  theory  surrounding  game-based 
training  focused  on  understanding  the  usefulness  of 
games  by  examining  motivation.  Training  researchers 
hypothesize  that  games  are  instructive  because  they 
engage  the  player/learner,  leading  to  skill  acquisition 
and  retention  (Colquitt,  LePine,  &  Noe,  2000;  Mathieu, 
Tannenbaum,  &  Salas,  1992;  Prensky,  2001; 
Tannenbaum  &  Yukl,  1992).  Empirical  research  has 
provided  evidence  that  attributes  of  games  (e.g., 
challenge,  realism,  and  interactivity)  influence  trainee 
motivation  and  the  length  of  time  that  trainees  are 
willing  to  invest  in  mastering  the  skills  taught  during 
game  play  (e.g.,  Corbeil,  1999;  Engel,  et  ah,  2009; 
Garris,  Ahlers,  &  Driskelt,  2002).  However,  research 
must  move  beyond  motivation  to  understand  how  to 
best  make  use  of  games  for  learning  and  training.  Just 
because  a  game  is  motivating  to  play  does  not  mean 
that  it  will  be  instructional.  In  fact,  what  is  motivating 
about  a  game  may  even  hinder  learning.  Research  has 
shown  that  if  the  content  is  not  correctly  embedded  into 
the  game,  learners  are  more  likely  to  walk  away  with 
increased  knowledge  of  how  to  play  the  game  rather 
than  a  transferable,  operational  knowledge  or  skill 
(Belanich,  Sibley,  &  Orvis,  2004). 
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Therefore,  there  is  a  need  for  additional  research  that 
leverages  sound  training  theories  and  practice  to  gather 
data  on  how  games  influence  the  acquisition  of  specific 
knowledge  and  skills,  Although  the  last  few  years  have 
brought  increased  research  and  theory  published  on  the 
use  of  serious  games,  there  is  still  much  work  to  be 
done  to  understand  if  and  how  games  can  be  effective 
training  tools.  The  existing  body  of  research  has  only 
started  to  provide  insight  as  to  how  well  game-based 
training  works  relative  to  other  methods,  and  why  or 
when  (under  what  conditions)  to  use  it.  Decisions  to 
use  games  to  train  have  been  based  on  a  “leap  of  faith,” 
and  there  have  been  few  efforts  to  test  that  faith 
through  rigorous  evaluation  of  the  hypothesized 
benefits  of  games  over  other  training  methods  (Hays, 
2005).  Training  designers  need  to  understand  how  to 
incorporate  game-based  approaches  to  provide  training 
that  is  pedagogicaily  sound,  as  well  as  motivating  and 
engaging.  The  impact  of  game-based  training 
programs  will  rise  dramatically  with  increased  solid 
empirical  research  conducted  on  the  effectiveness  of 
such  training  approaches. 

The  purpose  of  this  paper  is  to  use  scientifically  tested 
theories  of  training  design  and  evaluation  to  frame 
empirical  research  which  examines  the  general 
effectiveness  of  training  games.  In  addition,  the 
research  literature  indicates  that  training  effectiveness 
is  influenced  by  three  primary  determinants:  a)  the 
training  program  itself  (e.g.,  the  game),  b)  the  trainee 
(in  terms  of  personal  characteristics  and  attitudes),  and 
c)  the  situational  context  in  which  the  training  takes 
place  (Campbell  &  Kuncel,  2001;  Colquitt,  et  ah,  2000; 
Mathieu,  et  ah,  1992;  Salas  &  Cannon-Bowers,  2001). 
Therefore,  in  addition  to  studying  the  overall 
effectiveness  of  game-based  training,  this  research  also 
investigates  how  features  of  the  game  (specifically, 
game  difficulty),  as  well  as  situational  variables  (he., 
level  of  unit  preparation  prior  to  the  training  and  level 
of  leader  involvement  during  the  training)  impact 
training  outcomes.  Such  a  multi-pronged  approach 
helps  to  gather  data  not  only  on  //'games  are  effective 
training  tools  but  also  begins  to  explain  why  the  game- 
based  approach  may  or  may  not  be  effective.  By 
understanding  the  conditions  under  which  game-based 
training  may  be  more  or  less  effective,  suggestions  can 
be  made  for  improving  future  game-based  training 
endeavors. 

Training  Research  and  Hypothesis  Development 

This  paper  explores  two  areas  in  order  to  develop  the 
hypotheses  that  are  tested:  1)  why  should  games  make 
effective  training  systems?  And  2)  under  what 
conditions  can  games  offer  maximum  training 
effectiveness?  Both  of  these  questions  can  begin  to  be 


answered  by  first  examining  the  vast  literature  on 
training  effectiveness. 

Games  as  Trainers 

As  discussed  above,  common  thinking  about  the  use  of 
games  for  training  is  that  they  are  motivating  enough  to 
engage  the  trainee,  and  hence,  lead  to  learning. 
Training  motivation  is  certainly  a  precursor  to  any  type 
of  learning  and  should  not  be  neglected;  after  all,  if 
learners  are  not  motivated,  it  becomes  difficult  to  get 
them  to  even  pay  attention  to  the  material  being 
presented  (Noe  &  Colquitt,  2002).  However,  moving 
beyond  motivation,  there  are  a  number  of  reasons  why 
games  should,  and  can  be,  effective  trainers, 
Essentially,  games  have  several  inherent  attributes  that 
can  make  them  effective  training  systems.  A 
comprehensive  review  of  these  attributes,  and  how  and 
why  they  are  related  to  training  outcomes,  can  be  found 
in  Wilson,  et  al.  (2008).  In  this  work,  Wilson  and 
colleagues  synthesized  the  genera!  training  literature 
and  the  research  on  games  to  develop  a  matrix  linking 
18  gaming  attributes  (e.g.,  challenge,  assessment, 
control)  to  15  training  outcomes  (e.g.,  motivation, 
application,  declarative  knowledge).  The  attributes 
identified  by  Wilson,  et  ai.  can  be  associated  with 
general  principles  of  learning.  For  example,  greater 
amounts  of  control  present  within  a  game  indicate  that 
the  user  can  manipulate  and  adjust  more  things  within 
the  gaming  environment  and  hence  may  be  more 
actively  involved  in  the  training  game.  This  active 
involvement  should  promote  more  deliberate  cognitive 
processing,  and  ultimately  lead  to  more  learning 
(Kraiger,  2003).  As  another  example,  the  constant 
feedback  and  assessment  that  is  possible  through  a 
game  can  raise  the  self-efficacy  of  the  learner  so  he  or 
she  is  more  likely  to  exert  more  effort  and  keep  moving 
toward  goal  accomplishment  (Cannon-Bowers,  2010). 
This  research  illustrates  that  games  should  inherently 
be  effective  training  tools  (simply  based  on  their 
features).  However,  even  with  al!  of  this  theory,  there 
is  still  a  dearth  of  research  showing  empirical  evidence 
of  learning  occurring  through  game-based  training. 

Therefore,  this  research  first  sought  to  demonstrate  the 
general  effectiveness  of  game-based  training.  In  the 
current  study,  training  effectiveness  is  primarily 
defined  at  the  unit  level.  The  reason  for  examining  unit 
process  and  other  unit  level  variables  is  because  no 
matter  the  specific  technical  training  objective  that  was 
targeted,  the  game-based  training  studied  in  this 
research  required  individuals  to  work  together  to 
accomplish  their  mission.  There  was  a  focus  during  all 
training  events  on  communication,  learning  the  roles  of 
others,  and  working  together.  The  findings  presented  in 
this  paper  focus  on  training  effectiveness  as  defined 
according  to  the  impact  (from  pre-  to  post-training) 
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training  has  on  four  unit-level  variables  that  all  serve  as 
indicators  of  a  unit  working  well  together: 

Hypothesis  1:  Game-based  training  positively  impacts 
the  training  outcomes  of  a)  Unit  Process,  b)  Unit 
Cohesion,  c)  Unit  Efficacy,  and  d)  Unit  Effectiveness. 

Maximizing  Training  Effectiveness 

The  second  question  that  this  study  sought  to 
empirically  examine  concerned  how  to  maximize  the 
effectiveness  of  game-based  training.  Two  sets  of 
variables  were  examined:  1)  characteristics  of  the  game 
design,  and  2)  contextual  variables  surrounding  the 
training. 

In  terms  of  characteristics  of  the  game,  the  current 
study  focused  on  the  difficulty  of  the  training  scenario. 
Wilson  et  al  (2008)  identified  Challenge  as  one  of  the 
attributes  of  a  game  that  can  be  tied  to  specific  learning 
outcomes.  According  to  Wilson  et  al.,  this  attribute 
describes  the  genera!  difficulty  of  the  game  and 
“possesses  multiple  clearly  specified  goals,  progressive 
difficulty,  and  informational  ambiguity”  (p.  230).  The 
amount  of  challenge  or  difficulty  in  a  game  should 
prompt  trainees  to  acquire  new  knowledge  in  order  to 
overcome  the  difficulties  presented  in  the  game 
(Cannon -Bowers,  2010).  Therefore,  the  difficulty  of 
the  training  scenario  was  examined,  and  a  hypothesis 
concerning  how  the  difficulty  of  the  game  would  relate 
to  training  effectiveness  as  made.  In  general,  the 
difficulty  of  the  training  is  hypothesized  to  be 
positively  related  to  training  effectiveness: 

Hypothesis  2:  Difficulty  of  the  training  scenario 
positively  impacts  the  training  outcomes  of  a)  Unit 
Process,  b)  Unit  Cohesion,  d)  Unit  Efficacy,  and  d) 
Unit  Effectiveness. 

Although  games  possess  many  inherent  attributes,  such 
as  their  difficulty  level,  that  make  them  viable  training 
tools,  the  training  conditions  under  which  games  are 
used  may  not  always  be  optimal.  In  other  words,  the 
design  of  the  training  game  itself  is  not  enough  to 
result  in  learning.  The  situational  context  is  an 
important  piece  of  any  training  effectiveness 
evaluation,  including  when  using  games  as  training 
tools. 

One  of  the  fundamental  hypotheses  of  Noe’s  (1986) 
influential  work  on  training  effectiveness  is  that  pre- 
training  variables  influence  learning  during  training. 
Such  variables  can  enhance  or  negate  the  actual  impact 
that  the  training  has.  One  such  determinant  of 
effectiveness  is  organizational  support,  if  trainees  see 
that  the  organization  supports  and  values  the  training  in 


which  they  are  going  to  engage,  they  are  likely  to  take 
it  more  seriously  and  get  more  out  of  it.  The  same 
argument  can  be  made  for  the  involvement  of  leaders 
during  training  exercises.  Leaders  are  important  social 
influences  and  can  “positively  or  negatively  influence 
trainees’  motivation  for  training  or  their  perceptions  of 
the  utility  of  training”  (Kraiger,  2003,  p.  S73).  This 
research  surrounding  genera!  training  effectiveness  can 
and  should  be  applied  to  game-based  training  as  well, 
Even  though  the  importance  of  pre-training  variables 
has  been  well-documented,  when  it  comes  to  game- 
based  training,  it  seems  that  training  designers  and 
organizations  may  believe  that  the  game  itself  is 
enough  to  result  in  learning.  However,  these  contextual 
variables  must  be  taken  into  consideration  in  order  to 
maximize  training  effectiveness  using  games. 

The  current  research  examined  the  impact  of  two 
contextual  variables  on  training  effectiveness:  the  level 
of  unit  preparation  for  the  training  (i.e.,  how  much  did 
the  unit  prepare  for  the  training  beforehand),  and  the 
level  of  leader  involvement  during  the  training.  Both  of 
these  variables  can  serve  as  indicators  of  organizational 
and  leader  support.  A  higher  level  of  pre-training 
preparation  as  well  as  a  higher  level  of  leader 
involvement  may  encourage  the  trainees  to  take  the 
training  more  seriously  and  ultimately  get  more  out  of 
it.  Therefore,  we  have  made  several  hypotheses 
concerning  the  impact  of  these  contextual  variables  on 
various  measures  of  training  effectiveness: 

Hypothesis  3:  The  level  of  Unit  Preparation  for  the 
training  positively  impacts  the  training  outcomes  of  a) 
Unit  Process,  b)  Unit  Cohesion,  d)  Unit  Efficacy,  and 
d)  Unit  Effectiveness. 

Hypothesis  4:  Leader  Involvement  in  training 
positively  impacts  the  training  outcomes  of  a)  Unit 
Process,  b)  Unit  Cohesion,  d)  Unit  Efficacy,  and  d) 
Unit  Effectiveness. 

METHOD 

During  FY  2009,  the  Army  distributed  copies  of  VBS2: 
U.S.  Army  to  many  training  locations  all  over  the 
world.  With  such  an  aggressive  application  of  game- 
based  training,  there  was  a  need  to  evaluate  the 
effectiveness  of  the  game-based  training  platform.  One 
of  the  main  training  objectives  of  units  using  VBS2: 
U.S.  Army  is  convoy  training.  However,  no  matter  the 
specific  technical  objective,  things  like  crew 
coordination,  teamwork,  and  communications  were  a 
major  focus  of  all  training  sessions. 

It  is  important  to  note  that  the  research  described  here 
represents  a  field  study.  Therefore,  we  had  little  control 
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over  the  training  objectives  and  content.  As  a  result,  the 
measures  developed  had  to  be  generic  enough  to  span  a 
variety  of  training  objectives  chosen  by  the  units 
conducting  the  training. 

Training  Procedures 

Training  using  VBS2:  U.S.  Army  is  typically 
conducted  in  a  training  facility  that  contains  individual 
computers  for  each  Soldier  completing  the  training. 
After  the  training  unit  entered  the  training  facility,  each 
Soldier  was  assigned  a  different  role  or  position  for  the 
convoy  scenario  -  driver,  gunner,  truck  commander 
(TC),  or  ohserver,  One  Soldier  was  also  assigned  the 
position  of  convoy  commander.  In  addition,  the  role  of 
higher  headquarters  is  filled  by  either  a  training 
facilitator  or  someone  in  a  leadership  position  (e.g., 
company  .commander)  within  the  unit.  By  having  all  of 
these  roles  filled  during  the  training,  the  Soldiers  are 
able  to  get  a  sense  of  what  each  role  does,  learn  how  to 
utilize  each  position,  and  also  learn  to  convey  the 
appropriate  information  to  the  appropriate  person. 

Before  beginning  actual  training  missions,  Soldiers 
were  given  initial  training  on  how  to  use  the  game  in 
order  to  get  all  trainees  comfortable  with  the  controls. 
Following  this  initial  training  period,  the  training  unit 
was  given  its  mission,  and  planning  for  the  mission 
commenced.  All  Soldiers  were  then  briefed  on  the 
mission  (typically  by  the  convoy  commander)  and 
given  time  to  ask  questions.  Following  the  mission 
brief,  Soldiers  then  went  through  the  mission. 
Throughout  the  mission  scenario,  training  facilitators 
played  the  role  of  the  opposing  forces  (OPFOR)  and 
embedded  attacks  in  the  form  of  insurgents,  snipers, 
and  improvised  explosive  devices  (lEDs)  into  the 
scenario.  The  unit  had  to  achieve  their  mission 
objectives  while  contending  with  attacks  from  the 
OPFOR.  During  training  missions,  it  was  not 
uncommon  for  a  Soldier's  avatar  to  experience  a 
casualty  during  the  scenario.  All  of  these  elements 
added  a  sense  of  stress  to  the  training  mission, 
encouraging  the  Soldiers  to  take  the  training  seriously. 
In  addition,  the  complexity  of  the  attacks  by  the 
OPFOR  differed  from  one  training  mission  to  another. 
Therefore,  there  was  variance  in  the  level  of  challenge 
and  difficulty  present  in  the  missions. 

Finally,  after  the  mission  was  complete,  a  thorough 
after-action  review  (AAR)  was  conducted.  Across  the 
observed  missions,  there  was  variance  in  who  led  and 
conducted  the  AAR.  For  some  units,  the  AAR  was  led 
by  a  higher-level  leader  who  had  not  actually 
participated  in  the  training.  For  other  units,  it  was  led 
by  the  individual  given  the  position  of  Convoy 
Commander  during  the  training,  and  hence,  was 
deemed  “the  leader”  of  the  exercise.  In  other  instances, 


the  AAR  was  conducted  hy  a  Soldier  who  had  been 
selected  to  conduct  the  AAR  at  the  completion  of  the 
exercise.  Finally,  for  some  units,  the  AAR  was  led  by 
one  of  the  training  facilitators  and  not  hy  a  Soldier  or 
higher-level  leader  in  the  training  unit.  This  variance 
contributed  to  some  AARs  being  conducted  more 
thoroughly  than  others. 

Measures 

Most  measures  were  given  pre-  and  post-training.  The 
difference  between  the  two  administrations  was  in  the 
question  stems.  The  post-training  items  asked  the 
trainees  to  respond  to  the  items  now  that  they  have 
engaged  in  training  using  VBS2:  U.S.  Army, 

Several  measures  were  administered  to  assess  unit  level 
outcomes.  Examining  these  measures  over  time 
provides  evidence  of  the  training  impact  on  unit 
outcomes,  and  hence,  training  effectiveness.  These  unit 
level  variables  included  the  following:  Unit  Process 
measure  (13  items),  Unit  Cohesion  measure  (task  and 
interpersonal  cohesion  for  a  total  of  1 1  items).  Unit 
Efficacy  measure  (7  items),  and  overall  Unit 
Effectiveness  measure  (3  items).  All  of  these  measures 
were  administered  both  pre-  and  post-training.  More 
information  about  each  of  these  measures  is  in  Table  1 . 

In  addition,  several  individual  level  outcomes  were 
assessed:  Skill  Preparedness  (13  items  -  How  prepared 
do  you  feel  to  engage  in,..),  Task  Performance  (15 
items  -  Please  rate  your  ability  to  perform  each  of  the 
following  tasks  in  the  simulator...),  and  Training 
Motivation  (10  items).  Due  to  variance  in  training 
objectives,  however,  these  individual  level  outcomes 
were  more  difficult  to  measure.  Therefore,  this  paper 
focuses  on  presenting  in-depth  results  associated  with 
the  unit  level  outcomes;  however,  the  general  patterns 
of  results  that  emerged  with  the  individual  level 
variables  are  also  discussed. 

In  order  to  understand  the  conditions  under  which 
game-based  training  may  be  more  or  less  effective,  we 
also  generated  several  measures  to  assess  variables  that 
may  impact  the  effectiveness  of  the  training.  These 
measures  were  completed  by  outside  observers  to  the 
training. 

First,  the  training  facilitators  completed  measures 
regarding  the  difficulty  of  the  training  and  the  level  of 
unit  preparation  for  the  training.  Both  of  these 
measures  were  completed  at  the  end  of  each  mission 
completed  by  a  unit.  For  Mission  Difficulty,  training 
facilitators  assessed  the  difficulty  of  the  training 
mission  according  to  three  categories:  the  number  and 
intensity  of  the  OPFOR,  the  number  and  intensity  of 
IED  attacks,  and  the  stability  of  the  mission  (e.g.,  how 
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much  did  the  mission  objectives  change  during  the 
mission).  Each  of  these  three  characteristics  was  rated 
on  a  5-point  scale,  ranging  from  very  easy  to  very 
difficult.  The  facilitators  were  given  examples 
associated  with  each  of  the  three  categories  in  order  to 
guide  tbeir  ratings,  In  order  to  obtain  one  rating  of 
mission  difficulty,  the  ratings  were  averaged  across 
categories  and  across  facilitators  if  more  than  one 
facilitator  rated  the  level  of  mission  difficulty. 


Table  1.  Training  Effectiveness  Measure  Descriptions 


Measure 

Measure  Description 

Example  Itcin(s) 

Unit  Process 

Assess  Soldier  belief 
about  how  well  the  unit 
worked  together  during 
the  training  in  terms  of 
different  teamwork 
skills  (e.g., 
communication, 
monitoring  progress 
towards  goals). 

Research  has 
hypothesized  that 
engagement  in 
effective  unit  processes 
are  directly  linked  to 
increased  unit 
performance  (e.g., 

Marks,  Mathieu,  & 
Zaccaro,  2001). 

“My  training  unit 
members  and  I 
understand  how 
one  another  prefer 
to  communicate 
information  to 
other  members.” 
“My  training  unit 
members  and  I 
understand  how 
members  are 
going  to  work 
together  to 
achieve  our 
goals.” 

Unit 

Cohesion 

Assess  Soldier 
attraction  to  the  unit, 
including  their  tasks 
(task  cohesion)  and 
other  group  members 
(interpersonal 
cohesion).  Research 
has  shown  that  unit 
cohesion  influences 
unit  performance  (e.g. 
Gully,  Devine,  & 
Whitney,  1995). 

Task  Cohesion: 
“When  one  person 
is  struggling  with 
a  task,  another 
member  of  (he 
training  unit  will 
step  in  to  help.” 
Interpersonal 
Cohesion :  “Tile 
members  of  my 
training  unit  get 
along  with  one 
another.” 

Unit 

Efficacy 

Assess  Soldier 
confidence  level  in 
ability  of  the  unit  to 
work  together  and 
perforin  well  during 
the  training.  Research 
has  shown  a  positive 
relationship  between 
unit  efficacy  and 
performance  (e.g. 
Stajkovic,  Lee,  & 

Nyberg,  2009). 

“I  am  confident 
that  the  members 
of  my  training  unit 
and  I  can 
effectively  set 
contingency 
plans.” 

Unit 

Effectiveness 

Assess  Soldier  belief 
about  how  effective  the 
unit  was  during  the 
training  mission 

“This  training  unit 
is  effective.” 

The  measure  to  assess  the  level  of  Unit  Preparation  was 
designed  to  account  for  differences  in  how  much  effort 
the  unit  leaders  put  into  preparing  for  the  VBS2:  U.S, 
Army  training  prior  to  coming  to  the  training  facility. 
Differences  may  exist  in  terms  of  how  much  the  unit 
communicates  with  the  training  facilitators  ahead  of 
time  in  regard  to  mission  objectives,  training  scenarios, 
etc.  For  this  measure,  facilitators  rated  each  unit  on  a  5- 
point  scale  (1  ~  Completely  Unprepared;  5  = 

Completely  Prepared)  based  on  behaviors  such  as 
having  discussions  with  the  training  facilitators 
regarding  training  objectives  and  mission  design.  These 
ratings  were  averaged  across  facilitators  if  more  than 
one  facilitator  rated  the  level  of  preparation.  In  order  to 
classify  the  units  into  more  meaningful  groups,  a 
dichotomous  variable  that  classified  those  units  who 
received  an  average  rating  of  3  or  less  as  “Less 
Prepared”  and  those  units  who  received  an  average 
rating  of  over  3  as  “More  Prepared.”  This  dichotomy 
was  used  throughout  all  analyses  involving  this 
variable,  Although  the  ratings  on  this  measure  were 
based  on  interactions  between  the  training  facilitators 
and  unit  leadership,  it  is  thought  that  if  leaders  put 
more  time  into  thinking  about  and  discussing  specific, 
training  objectives  with  the  facilitators,  this  level  of 
preparation  would  have  been  conveyed  to  the  unit 
members,  and  hence,  the  unit  as  a  whole  would  have 
spent  more  time  preparing  for  the  training  activity. 

Training  observers  on  the  project  research  team 
completed  the  Leader  Involvement  measure  as  each 
training  session  was  observed.  This  measure  consisted 
of  six  questions  targeted  at  recording  the  level  of 
involvement  from  unit  leadership  (not  actually 
involved  as  a  participant  in  the  training)  during  each 
training  mission.  The  qualitative  information  from  each 
of  these  measures  was  transformed  to  a  quantitative 
three-point  scale  to  measure  the  level  of  involvement, 
A  rating  of  1  meant  that  unit  leadership  was  not  present 
at  the  training  exercise  (no  leader  involvement);  a 
rating  of  2  meant  that  unit  leadership  participated  in 
either  the  mission  brief  or  the  AAR  portion  of  training 
(some  leader  involvement);  and  a  rating  of  3  meant  that 
unit  leadership  was  present  and  participating  in  both  of 
those  important  aspects  of  the  training  (substantial 
leader  involvement). 

Participants 

One-hundred  and  sixty  five  Soldiers  at  two  military 
installations  participated  in  the  evaluation  study.  These 
165  Soldiers  represented  9  different  platoons,  95%  of 
these  Soldiers  were  enlisted;  4%  were  officers;  and  less 
than  1%  were  warrant  officers.  The  mean  age  of  the 
trainees  was  25.20  (SD  =  5.68).  In  addition,  33%  of  the 
trainees  reported  having  engaged  in  training  using 
VBS2:  U.S.  Army  in  the  past. 
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Upon  Soldiers  entering  the  training  facility,  a  brief 
overview  of  the  project  was  given.  Soldiers  were 
informed  that  their  participation  was  voluntary  and  that 
all  of  their  responses  were  anonymous  and  would  not 
be  linked  back  to  any  single  individual.  Soldiers  then 
read  and  signed  the  informed  consent  form,  and  then 
were  given  the  pre-training  questionnaire  packet  to 
complete.  This  set  of  measures  took  approximately  15 
to  20  minutes  for  each  Soldier  in  the  unit  to  complete. 
Following  completion  of  these  measures,  the  unit 
engaged  in  training  as  usual  using  VBS2:  U.S.  Army. 
Soldiers  completed  the  post-training  measures  after  the 
AAR. 

RESULTS 

Hypothesis  1  was  generated  to  demonstrate  the  general 
effectiveness  of  game-based  training  by  focusing  on 
unit  level  outcomes  and  predicted  a  positive  impact  of 
the  game-based  training  on  four  unit  level  outcomes: 
Unit  Process,  Unit  Cohesion,  Unit  Efficacy,  and  Unit 
Effectiveness.  Out  of  the  four  unit  level  outcomes 
measured,  reported  levels  of  Unit  Process  and  Unit 
Cohesion  significantly  increased  from  pre-  to  post¬ 
training,  Although  not  significant,  perceived  levels  of 
Unit  Efficacy  and  Unit  Effectiveness  trended  in  the 
predicted  direction.  These  results  are  displayed  in 
Table  2, 


Table  2,  Pre-  to  Post-Training  Comparisons  on  Unit 
Level  Variables 


Comparison 

M 

SD 

1 

Process 

Pre 

3.30 

.70 

-2.71  * 

Post 

3.46 

.78 

Cohesion 

Pre 

3.41 

.68 

-1.93* 

Post 

3.54 

.89 

Efficacy 

Pre 

3.46 

.78 

-1.33 

Post 

3.55 

.82 

Effectiveness 

Pre 

3,07 

.78 

-1.40 

Post 

3.16 

.83 

*  p  <..05 

The  results  from  these  analyses  provide  some  evidence 
of  general  training  effectiveness.  The  remainder  of  the 
hypotheses  focused  on  the  conditions  under  which 
game-based  training  can  result  in  maximum 
effectiveness.  First,  hypothesis  2  focused  on  the 
difficulty  of  the  game-based  training  scenarios  and  the 
impact  of  that  training  characteristic  on  training 
effectiveness.  No  parts  of  this  hypothesis  were 


supported;  the  difficulty  of  the  mission  scenarios  did 
not  significantly  impact  any  of  the  training 
effectiveness  outcomes.  Analyses  did  demonstrate, 
however,  that  after  accounting  for  pre-training  levels  of 
skill  preparedness,  Mission  Difficulty  significantly 
impacted  post-training  levels  of  Skill  Preparedness  ((3  = 
.16,  AR2  =  .03,  p  <  .05).  These  results  indicate  that 
while  the  difficulty  of  the  mission  did  not  have  any 
impact  on  the  unit  level  training  effectiveness 
outcomes,  trainees  felt  more  prepared  to  apply  certain 
skills  (e.g.,  assessing  a  tactical  situation)  following  a 
more  difficult  training  mission. 

Hypotheses  3  and  4  focused  on  the  impact  of 
contextual  variables  (Unit  Preparation  and  Leader 
Involvement)  on  training  effectiveness.  All  of  these 
hypotheses  were  tested  using  an  Analysis  of 
Covariance  (ANCOVA)  to  control  for  pre-training 
levels  of  the  outcome  variables. 

In  general,  results  demonstrate  that  both  contextual 
variables  impact  game-based  training  effectiveness. 
Specifically,  all  parts  of  Hypothesis  3  were  supported; 
trainees  in  units  with  less  preparation  reported  lower 
levels  on  all  four  outcomes  compared  to  trainees  in 
units  with  more  preparation:  a)  Unit  Process  (Ma(jj  = 
3.34  vs.  Ma{[j  =  4.00,  F(l,  126)  =  10.23,  rf  =  .08,  p  < 
.01),  b)  Unit  Cohesion  (Ma(Jj  =  3,43  vs.  Madi  =  4.05, 
F(i,  126)  -  6.72,  rf  =  .05,  p  <  .01),  c)  Unit  Efficacy 
(Madj  =  3.46  vs.  Madj  =  3.92,  F(l,  125)  =  3.95,  q2  =  .03, 
p  <  .05),  and  d)  Unit  Effectiveness  =  3 .07  vs.  Madj 
=  3.60,  F(l,  125)  =  5^00,  if  =  .04,  p  <  .05).  Figure  1 
illustrates  the  impact  of  Unit  Preparation  on  Unit 
Process.  The  results  from  the  other  parts  of  the 
analyses  all  follow  the  same  pattern. 


Less  Prepared  More  Prepared 

Unit  Preparation 

*p<.OI 


Figure  1 .  Impact  of  Unit  Preparation  on  Unit  Process. 

In  terms  of  the  impact  of  Leader  Involvement  on 
training  effectiveness  (Hypothesis  4),  significant 
relationships  existed  only  with  Unit  Process.  Trainees 
in  units  with  increasing  levels  of  leader  involvement 
during  the  training  reported  higher  levels  of  unit 
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process,  after  adjusting  for  pre-training  levels  of  the 
outcome  variable,  F(2,  137)  =  4.02,  q2  =  .06,  p  <  .05. 
Figure  2  displays  the  results  of  this  analysis. 


No  Leader  Some  Leader  Substantial 
Involvement  Involvement  Leader 

Involvement 


Level  of  Leader  Involvement  during  Training 

*  Adjusted  means  significantly  different  from  one 
another,  p  <  .05 

Figure  2.  Impact  of  Leader  Involvement  on  Unit 
Process. 

Results  examining  the  impact  of  Unit  Preparation  and 
Leader  Involvement  on  the  individual  level  outcomes 
showed  a  similar  pattern  of  results.  Trainees  in  units 
with  less  preparation  reported  significantly  lower  levels 
on  all  three  outcomes  (Skill  Preparedness,  Task 
Performance,  and  Training  Motivation)  compared  to 
trainees  in  units  with  more  preparation.  In  addition, 
trainees  in  units  with  increasing  levels  of  Leader 
Involvement  during  the  training  reported  significantly 
higher  levels  of  task  performance  and  training 
motivation,  after  adjusting  for  pre-training  levels  of  the 
outcome  variables. 

DISCUSSION  AND  RECOMMENDATIONS  FOR 
FUTURE  RESEARCH 

The  purpose  of  this  research  was  to  collect  empirical 
data  on  the  general  effectiveness  of  serious  games,  as 
well  as  on  the  characteristics  and  conditions  that  may 
help  to  maximize  the  effectiveness  of  game-based 
training.  This  paper  was  intended  to  begin  to  apply 
some  of  the  sound  science  and  rigor  that  characterizes 
training  effectiveness  studies  to  game-based  training. 

In  this  paper,  training  effectiveness  focused  on  four 
unit  level  outcomes:  Unit  Process,  Unit  Cohesion,  Unit 
Efficacy,  and  Unit  Effectiveness.  Although  the  increase 
from  pre-  to  post-training  levels  was  not  statistically 
significant  for  all  of  these  variables,  all  trended  in  the 
predicted  direction,  with  trainees  reporting  lower  levels 
of  these  variables  pre-training  as  compared  to  post. 
These  results  demonstrate  the  general  effectiveness  of 
game-based  training. 


Perhaps  more  informative,  however,  are  the  results  of 
the  other  analyses  looking  at  the  impact  of  game 
characteristics  (specifically,  Mission  Difficulty)  and 
the  context  of  the  training  in  relation  to  the  training 
effectiveness  outcomes.  First,  the  difficulty  of  the 
training  did  not  have  an  impact  on  any  of  the  unit  level 
training  effectiveness  outcomes.  Even  though  the 
difficulty  of  a  game  is  purported  to  build  engagement 
within  the  players  in  terms  of  encouraging  them  to 
continue  trying  to  overcome  the  challenges,  this  line  of 
theory  may  not  apply  to  the  unit  level  outcomes 
examined  in  this  paper.  Perhaps  as  the  training 
missions  got  more  difficult,  the  individual  trainees 
become  more  focused  on  the  technical  portion  of  the 
game,  and  focused  less  on  communicating  with  their 
unit  members,  for  example.  This  theorizing  is  backed 
up  to  some  extent  by  Mission  Difficulty  positively 
influencing  the  Skill  Preparedness  of  trainees  following 
the  training.  This  result  illustrates  that  perhaps  the 
influence  of  mission  difficulty  is  not  so  much  on  how  a 
team  works  together,  but  more  on  individual  skill 
acquisition  and  feelings  of  readiness.  In  addition,  there 
may  he  moderator  variables  masking  the  impact  of 
training  difficulty  on  training  effectiveness  at  the  unit 
level.  For  example,  perhaps  mission  difficulty  only 
impacts  training  effectiveness  for  those  trainees  that 
had  some  degree  of  experience  using  the  game-based 
training  program.  It  is  important  to  match  the  degree  of 
difficulty  and  challenge  in  any  training  program  to  a 
level  appropriate  for  the  trainees.  This  relationship 
should  continue  to  be  investigated  in  future  research. 

The  final  set  of  analyses  presented  in  this  paper 
demonstrated  the  importance  of  taking  situational 
factors  into  account  when  using  game-based  training. 
Specifically,  the  results  reported  here  demonstrated  the 
impact  of  Unit  Preparation  for  Training  and  Leader 
Involvement  on  training  effectiveness  (for  both  unit 
and  individual  level  outcomes).  Soldiers  in  units  that 
were  more  prepared  for  the  training  reported  higher 
levels  of  Unit  Process,  Unit  Cohesion,  Unit  Efficacy, 
and  Unit  Effectiveness  following  the  training  compared 
to  Soldiers  in  units  with  lower  levels  of  preparation.  It 
is  likely  that  units  that  spent  time  preparing  for  the 
training  exercises  emphasized  and  focused  on  getting 
the  unit  to  work  together.  This  emphasis  was  reflected 
in  the  results.  In  addition,  Unit  Preparation  had  a 
positive  impact  on  all  three  individual  level  outcomes. 
Therefore,  it  appears  that  units  who  put  more  time  into 
preparing  for  the  training  also  passed  on  training 
information  to  unit  members,  perhaps  focused  on 
taking  the  training  seriously  and  also  about  how  to  get 
more  out  of  the  training.  It  may  also  be  the  case  that  the 
units  with  the  higher  preparation  levels  had  more  well- 
defined  training  objectives  which  helped  unit  members 
get  more  out  of  the  training.  Whatever  the  case  may  be. 
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unit  preparation  prior  to  the  training  seems  to  be  a 
powerful  antecedent  to  game-based  training 
effectiveness. 

In  contrast  to  the  influence  of  Unit  Preparation  on  all 
four  unit-level  variables,  the  level  of  Leader 
Involvement  in  the  training  only  influenced  the  level  of 
Unit  Process  reported  by  the  trainees,  Understanding 
the  nature  of  this  involvement  may  help  to  explain  why 
process  was  the  only  variable  influenced.  When  unit 
leadership  was  involved  during  the  training,  their 
involvement  likely  came  in  the  form  of  feedback  about 
how  the  unit  was  working  together  (i.e.,  their  team 
processes)  and  was  not  centered  on  motivational  states 
or  overall  performance  feedback.  Future  research 
should  look  at  this  variable  in  closer  detail  to 
understand  what  exactly  it  is  about  leader  involvement 
that  potentially  makes  it  an  important  variable.  For 
example,  is  it  enough  for  leaders  to  just  be  present 
during  training,  or  do  they  need  to  provide  a  certain 
type  of  feedback  to  be  influential?  What  is  clear  from 
these  results,  however,  is  that  the  context  in  which 
game-based  training  is  embedded  is  not 
inconsequential.  Training  designers  must  carefully 
consider  the  surrounding  environment  when  deciding 
to  make  use  of  game-based  training.  By  making  sure 
that  trainees  feel  supported  during  the  training,  the 
effectiveness  of  that  training  is  likely  to  he  maximized. 

In  general,  the  results  of  this  study  illustrate  the 
importance  of  going  beyond  assuming  that  games  are 
effective  training  tools  because  they  are  motivating  and 
engaging  for  the  trainee.  In  order  to  further  the  use  of 
games  for  training,  principles  generated  under  general 
training  effectiveness  research  must  be  utilized, 
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